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ABSTRACT 

This study investigates the effects on various learning 
outcomes of an inquiry-based multimedia learning environment called Astronomy 
Village: Investigating the Universe. Students were pre-/post-tested on their 
conceptual understanding, well-structured problem solving abilities, and ill- 
structured problem solving abilities. The results of the study indicate that 
Astronomy Village is an effective tool for helping students learn important 
astronomy content and well-structured problem-solving skills. The inquiry- 
based approach also provides for a more robust understanding of the problem 
so that concepts and problem-solving strategies can be applied in a new 
context. This study also reports that there are significant implications to 
be drawn from the lack of transfer to ill-structured problems. (KHR) 
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Introduction 

In the science education community, educators have put forth considerable effort 
towards improving students’ problem-solving abilities (National Research Council, 1996). 
Instructional strategies that engage students in scientific inquiry offer great promise for 
enhancing student problem solving in science (Geban, Askar, & Ozkan, 1992). These 
inquiry-based instructional strategies emphasize important aspects of scientific problem 
solving, namely identifying questions to investigate, designing investigations, conducting 
investigations, formulating conclusions and communicating results (Robitai lie et. al., 

1993). Multimedia learning environments provide a useful vehicle for engaging students in 
scientific inquiry. They make it possible to expose students to problem-solving experiences 
that are difficult to create in classroom situations (Geban, et al., 1992; Zietman & Hewson, 
1986). 

Many studies have shown that inquiry-based instructional strategies produce 
significantly greater science achievement and problem solving than conventional 
instructional strategies (Anderson, DeMelo, Szabo, & Toth, 1975; Gabel et al., 1977; 

Tobin & Capie, 1982, Bredderman, 1983; Szymansky, Kyle, and Alport, 1983; Cavin & 
Lagowski 1 978; Bolick, 1972; Hughes, 1974). Unfortunately, the measures of 
achievement and problem solving for most of these studies provide only a broad-scale 
description of learning. Without fine-grained measures of learning it becomes difficult to 
investigate how students develop problem-solving abilities. 

Researchers have argued that problem solving is a complex skill that is influenced 
by a set of related skills, namely, conceptual understanding in a domain, general 
metacognitive abilities, and attitudes towards science (Hong, 1 998). The extent to which 
these related skills contribute towards successful problem solving depends on the nature of 
the task. In one study, a dichotomy between well-structured and ill-structured problem 
solving was achieved through the development of problem-solving assessments in 
astronomy (Hong, 1998). In general, ill-structured problems have more than one valid 
solution and can be solved in more than one way (Simon, 1973). In contrast, well- 
structure problems have one correct answer and typically only one way to reach a final 
solution (Simon, 1978 & 1979). Using the astronomy assessment instruments, researchers 
found that the largest predictor of well-structured problem-solving performance was 
conceptual understanding (Hong, 1998). However, for ill-structured problems, 
metacognitive ability and attitude towards science also became important predictors of 
problem-solving performance. 

The purpose of this research study is to investigate the effects on various learning 
outcomes of an inquiry-based, multimedia learning environment called Astronomy Village®: 
Investigating the Universe ™. The students were pre/post tested on their conceptual 
understanding, well-structured problem solving abilities, and ill-structured problem solving 
abilities. 



1 For further information, please contact: Steven McGee, 316 Washington Avenue, Wheeling, WV 26003, (304) 243-2388, 
incgee@cet.edu 
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Methods 



Subjects 

The participants in this study were I 19 9th-grade students attending a high school 
in a small working-class community near a large, Midwestern city. All 9th-grade students 
enrolled in the earth and space science course were invited to participate. 

Procedure 

Astronomy Village uses the metaphor of living and working at a mountain-top 
observatory (the village) as the primary interface from which students investigate 
contemporary problems in astronomy (see Pompea and Blurton, 1995). The environment 
includes ten investigations covering a broad cross-section of current research areas in 
astronomy. In this study, students used Astronomy Village for three weeks. The teacher 
randomly assigned students to conduct one of two Astronomy Village investigations, 
Nearby Star or Variable Star. In the Nearby Star investigation students use the concept of 
parallax to measure the distance to nearby stars. In the Variable Star investigation, students 
use the inverse square law to determine the distance to a nearby galaxy. 

For each investigation, students progressed through five phases of research: 
background research, data collection, data analysis, data interpretation and presentation of 
results. In the background research phase, students collected information by reading 
articles in the virtual library and listening to lectures in the virtual theatre. In the data 
collection phase, students went to the virtual observatory to collect images necessary for 
their investigation. In the data analysis and data interpretation phases, the students analyzed 
their images and interpreted the results of this analysis. Finally, each student team 
presented its results to the class. During the investigation, students used an electronic 
LogBook for recording their scientific notes and observations during their investigation. 

Instruments 

There were four assessment instruments used for this study— two well-structured 
problems and two ill-structured problems (see Hong, 1998 for a description of how the 
instruments were developed). One well -structured problem focused on extensions to the 
Nearby Star investigation and the other one focused on extensions to the Variable Star 
investigation. The ill-structured problems were developed to assess students’ problem- 
solving skills using two novel problems that do not have a single correct solution. One of 
the two problems is similar in content to the Nearby Star and Variable Star investigations. It 
is called a “less-structured” problem. This problem asks students to propose how they 
would determine whether an incoming asteroid was in danger of hitting Earth. The other ill- 
structured problem requires more structural understanding, such as scientific principles and 
processes, than content understanding in order to reach a successful solution. In this 
problem, called an “ill -structured” problem, students had to consider alternative, competing 
goals by proposing a site for a new telescope from three alternatives. For each well- 
structured problem students were asked to classify important concepts related to the 
problem as a measure of their concept understanding in the domain. 

Before students used Astronomy Village, the teacher and an investigator spent two 
classroom periods administering the problem-solving tests. There was no time limitation 
and students were permitted to spend as much time as necessary to solve the problems and 
answer the questionnaires. After finishing Astronomy Village, a posttest was administered 
during two classroom periods. In the both pre- and post-tests, each student took all of the 
tests, which included conceptual understanding, the Nearby Star well-structured problem, 
the Variable Star well-structured problem, and the two ill-structured problems. Two raters 
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scored students’ responses individually, based on an agreed upon scoring systems. The 
overall inter-rater reliability was .82. 

Results 

In the first analysis we looked at the impact of Astronomy Village on conceptual 
understanding and well-structured problem solving. Since students conducted only one 
investigation (either Nearby Star or Variable Star) and they took both well-structured tests, 
the test that is based on the path that they did not complete serves as a within subjects 
control. Table I presents the means and standard deviations of the dependent variables for 
both groups and pre- and post-test. 





Conceptual Understanding 


Well-Structured Problem Solving 


Groups N 


Pretest Posttest 


Pretest Posltesl 


Treatment 119 

Control 119 


1.09 (.928) 2.46(1.34) 

1 .25 (.989) 1 .46 ( 1 . 1 3) 


.21 (.41) 1.45(1.41) 

.27 (.40) .39 (.69) 



Table I: Comparison of Pretest and Posttest means between the treatment and control (standard 
deviations are show in parentheses) 

A two-way analysis of variance was employed to analyze the differences between 
students’ improvement on conceptual understanding and well-structured problem solving. 
The results revealed a statistically significant interaction effect for both conceptual 
understanding (F = 32.36, p = .000) and for well-structured problem solving (F = 53.95, 
p < .000). At pretest time, the treatment scores were significantly lower than the control 
scores for both content understanding and well-structured problem solving. However, the 
treatment scores improved statistically from pre- to posttest and were statistically higher 
than the control scores on both content understanding and problem solving. 

Finally, Table 2 shows the means and standard deviations of pretest and posttest 
differences on the less- and ill-structured problems. Using a t-test, no statistically 
significant differences were found between the mean scores of the pretest and those of the 
posttest for either the less-structured (t = ,60, p = .55) or ill-structured (t = .65, p = .5 1) 
problem-solving tests. 





Less-Structured Problem Solving 




Ill-Structured Problem Solving 




N M 


SD 


N 


M SD 


Pretest 


117 3.60 


3.09 


1 17 


4.59 1.96 


Posttest 


114 3.35 


3.28 


115 


4.42 2.17 



Table3: Comparison of pretest and posttest differences on the ill-structured problems (standard 
deviations are shown in parentheses) 



Discussion and Implications 

The results of this study indicate that Astronomy Village is an effective tool for 
helping students learn important astronomy content and well-structured problem-solving 
skills. Since the well-structured problems on the assessment instrument were slightly 
different than the problems encountered in the software, students had to adapt their 
understanding of parallax or the inverse square law to successfully solve the problems. The 
authors argue that having students engage in image analysis activities that are analogous to 
the techniques that scientist use provides students with a better understanding of the 
underlying concepts. The inquiry-based approach also provides for a more robust 
understanding of the problem so that concepts and problem-solving strategies can be 
applied in a new context. 
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However, the results of this study also indicate that there is a limitation in how far 
this knowledge will transfer. Students were not able to generalize the process of scientific 
inquiry to other areas of astronomy that were not directly related to the investigations that 
they conducted. There are two significant implications that can be drawn from the lack of 
transfer to ill-structured problems. 

The first implication is that more work needs to be done on Astronomy Village so 
that it will better support the process of scientific inquiry. Students using Astronomy 
Village typically take from 3-4 weeks of class time to complete one investigation. Unless 
Astronomy Village is used in the context of an astronomy course, teachers typically have 
students conduct only one investigation (McGee, Howard, & Hong, 1998). This does not 
provide much opportunity for students to generalize about the nature of scientific inquiry 
across multiple investigations. Astronomy Village provides scaffolding for students to 
complete an investigation, but there is little opportunity for fading as prescribed in the 
cognitive apprenticeship model (Collins, Brown, and Newman, 1989), which would allow 
students to learn more about the general processes of problem solving. In a subsequent 
version of Astronomy Village , designers have created opportunities for students to pursue 
one topic across multiple investigations (McGee & Howard, 1999). Future research will 
investigate whether this improves the students’ ability to generalize the processes of 
scientific inquiry. 

The second implication is that this study provides further evidence that there are 
differences in how students approach well- vs. ill-structured problems. In Astronomy 
Village , the investigations presented students with well-structured problems, whereas the 
problems that students did not improve on were ill-structured, which students were never 
exposed to during Astronomy Village. Since the students were accustomed to only well- 
structured problems, it is possible that they might have used well-structured strategies in 
their attempts to solve the ill-structured problems. This result supports other research that 
argues that well- and ill-structured problems engage different problem-solving processes 
(Sinnott, 1989; Voss & Post, 1 988; Voss, 198 I ; Wood, 1994; Reitman, 1965). 

There are three main areas for future research on well- vs. ill-structured problem 
solving. The first set of research questions should focus on the role of well-structured 
problem solving in solving ill-structured problems. A second set of questions to be 
explored relates to how students integrate new information into prior knowledge for 
solving problems. A third set of questions should focus on the role of multimedia learning 
environments, such as Astronomy Village , in helping students to develop scientific ill- 
structured problem-solving abilities. 
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